Four participants were invited to share their views as to what they see as the next steps for monitoring and evaluation in Africa RISING: Tahirou Abdoulaye (IITA, based in Nigeria), Gary Ender (Abt Associates), Carlo Azzarri (IFPRI) and Jerry Glover (USAid) all three based in the United States.
Tahirou:
We have made very good progress
We’re devoting 2.5 days on this which is usually put aside until the end of the program. We are making good progress, despite some issues to be discussed – quickly:
We need to have randomization for a good eval set up – how? In concrete terms? Let’s have written guidelines on what needs to be done and who will do it and how?
We can’t overemphasize the need to have M&E officers quickly in place
We need to involve some CG folks – CG centers have economists that might help us on the ground – what is clearly their role?
Gary:
I want to second Tahirou on the progress made on collaboration, evaluation design.
I need to focus on the short run: the sense I got on the narrow topic of feed the future is that we need some work to be done on data collection and submission – what’s planned in terms of who enters data in the system. Once we know what indicators are we need to collect and enter data etc. so someone will have to be ready to do that on those teams. Someone needs to check the quality.
Furthermore, about indicators, we had some tentative indicators for quick win activities etc. we need a process to define about longer term indicators – sthg to be done pretty soon to match the kind of results that AR wants to achieve. We need to focus on this for partnering with other projects.
How will USAid count that? There is potentially a duplicate reporting issue there. We need to think about it. The indicators I proposed might be relevant but it could be someone else that could take care of these indicators.
We need to come to a decision with USAid, IFPRI etc. about when we submit a draft M&E plan. We need a lot of info from you to finalise this plan. The content comes from all of you. We need to decide on timing and when is an appropriate time. Then we need to improve it etc.
Carlo:
I’m impressed by the progress over the last 2.5 days. I’m very happy this happened. On the specific M&E indicators we still need to resolve some issues. The process is now clear, and site selection is clear in as far as Chris will be working with Spatial Dev so implementers can access this data quite soon (next week) – so it will be easier to select final sites.
In terms of evaluation, I’m very happy to have narrowed down to 3 important eval questions. These are very clear questions. I can anticipate that I have the methods to address the first 2 questions – less so on the final one (scaling up) but I’ll discuss this with colleagues and with this group.
On the evaluation questions, we need data and qualitative and quantitative surveys.
On the questionnaires, for the quick start we didn’t implement them because it was too short but we have a draft questionnaire for ET and for Malawi (which can be adapted to Tanzania) and we have yet to design the questionnaire for West Africa.
In terms of sequencing, we have the evaluation questions and some methods etc. but the other ingredients we need are research activities clearly spelt out and clear number of sites/villages/farmers to target as beneficiaries. With this we can move on and design the evaluation.
About the resources we have, we cannot evaluate all the project in all the sites so we might select some sites in AR and provide a rigorous evaluation for those and if we want to keep the other sites open we can do it. We would have a twin track approach with some sites that might be or will be evaluated. Malawi is very advanced in evaluation design so we can make a lot of progress. In Tanzania, we should still think of an evaluation because the longer term sites are not clear yet. Then we have a question mark for Ghana, Mali and Ethiopia. On the former two we can’t cover the two. So I need to discuss with Stan and other colleagues and we might select sites from Ghana. For Mali Sibiry is taking care of the coordination of sites. We might select some sites in ET highlands too and randomize not the whole AR project.
If we are partnering with other institutes we should let those other projects do the evaluation and focus on other areas.
Jerry:
We have the cereal system initiative in South Asia which developed an M&E framework before the program started. There are lessons to be learned there. Stan is in touch with the CISSA M&E officer.
It’s necessary for a small group of us (Irmgard, Peter, Asamoah, Mateete, Carlo) to convene a meeting with the M&E people working on it to review our plan. (Carlo: Apparently there aren’t lessons to learn from that project). The meeting would be good.
Formulating the evaluation questions around sustainable intensification (about sustainably intensified systems) would be great. What questions could we focus on to formulate evaluation questions around SI principles.
The landscape scale characterization (e.g. vegetation etc.) is useful e.g. from a project by ICRAF funded by DfID – this could be another layer we could include at a later stage. It’s a high priority for me.
In Ibadan we briefly discussed ethics and we identified the need to include text on ethics (e.g. privacy, dealing with farmers and not raising expectations etc.), language we can work from etc. that would be great to have in the research program on M&E and other topics.
We didn’t get to data flow protocol but that’s not going to be an easy topic. In CISSA it was not fully resolved. The principles in that project could inform ways that scientists get incentives to share data. IFPRI will have to follow a bottom up process so that IFPRI and others get data in a timely manner.
The program document – we developed a ToC but I couldn’t find it anywhere. That will be our product for September 30. In there we can have timelines explaining how the M&E framework will be more fully developed. There will be a chapter and timeline dedicated to M&E. We need a living program document by September 30, covering these issues. If we haven’t made sufficient progress we need to explain how we will make progress.
Q: We need clear objectives, types of activities, types of expected results to finalize the program document.
A: Yes, the program doc will give the general approaches and framework and we’ll have further details for specific projects, including research and development hypotheses, outcomes, M&E framework, ILRI comms strategy etc.
The research framework is about how – but we now need to focus on the ‘what we do’.
Comments:
For me to help with Africa RISING, I need Mali to be part of the M&E work. There are constraints from the US Gov’t. Logistics is not a good reason to cut in Mali. Cut at thematic level in terms of the types of technologies etc. that you will evaluate, not on geo boundaries.
The questionnaire needs to be checked by the research team before it’s finalized.
Thank you for the level of efforts of everyone over the last 2.5 days.
Africa RISING M&E Expert Meeting
5-7 September 2012Large auditorium, ILRI Ethiopia, Addis Ababa
Back to the event agenda
Monitoring and Evaluation next steps
Four participants were invited to share their views as to what they see as the next steps for monitoring and evaluation in Africa RISING: Tahirou Abdoulaye (IITA, based in Nigeria), Gary Ender (Abt Associates), Carlo Azzarri (IFPRI) and Jerry Glover (USAid) all three based in the United States.Tahirou:
We have made very good progress
We’re devoting 2.5 days on this which is usually put aside until the end of the program. We are making good progress, despite some issues to be discussed – quickly:
Gary:
I want to second Tahirou on the progress made on collaboration, evaluation design.
I need to focus on the short run: the sense I got on the narrow topic of feed the future is that we need some work to be done on data collection and submission – what’s planned in terms of who enters data in the system. Once we know what indicators are we need to collect and enter data etc. so someone will have to be ready to do that on those teams. Someone needs to check the quality.
Furthermore, about indicators, we had some tentative indicators for quick win activities etc. we need a process to define about longer term indicators – sthg to be done pretty soon to match the kind of results that AR wants to achieve. We need to focus on this for partnering with other projects.
How will USAid count that? There is potentially a duplicate reporting issue there. We need to think about it. The indicators I proposed might be relevant but it could be someone else that could take care of these indicators.
We need to come to a decision with USAid, IFPRI etc. about when we submit a draft M&E plan. We need a lot of info from you to finalise this plan. The content comes from all of you. We need to decide on timing and when is an appropriate time. Then we need to improve it etc.
Carlo:
I’m impressed by the progress over the last 2.5 days. I’m very happy this happened. On the specific M&E indicators we still need to resolve some issues. The process is now clear, and site selection is clear in as far as Chris will be working with Spatial Dev so implementers can access this data quite soon (next week) – so it will be easier to select final sites.
In terms of evaluation, I’m very happy to have narrowed down to 3 important eval questions. These are very clear questions. I can anticipate that I have the methods to address the first 2 questions – less so on the final one (scaling up) but I’ll discuss this with colleagues and with this group.
On the evaluation questions, we need data and qualitative and quantitative surveys.
On the questionnaires, for the quick start we didn’t implement them because it was too short but we have a draft questionnaire for ET and for Malawi (which can be adapted to Tanzania) and we have yet to design the questionnaire for West Africa.
In terms of sequencing, we have the evaluation questions and some methods etc. but the other ingredients we need are research activities clearly spelt out and clear number of sites/villages/farmers to target as beneficiaries. With this we can move on and design the evaluation.
About the resources we have, we cannot evaluate all the project in all the sites so we might select some sites in AR and provide a rigorous evaluation for those and if we want to keep the other sites open we can do it. We would have a twin track approach with some sites that might be or will be evaluated. Malawi is very advanced in evaluation design so we can make a lot of progress. In Tanzania, we should still think of an evaluation because the longer term sites are not clear yet. Then we have a question mark for Ghana, Mali and Ethiopia. On the former two we can’t cover the two. So I need to discuss with Stan and other colleagues and we might select sites from Ghana. For Mali Sibiry is taking care of the coordination of sites. We might select some sites in ET highlands too and randomize not the whole AR project.
If we are partnering with other institutes we should let those other projects do the evaluation and focus on other areas.
Jerry:
We have the cereal system initiative in South Asia which developed an M&E framework before the program started. There are lessons to be learned there. Stan is in touch with the CISSA M&E officer.
It’s necessary for a small group of us (Irmgard, Peter, Asamoah, Mateete, Carlo) to convene a meeting with the M&E people working on it to review our plan. (Carlo: Apparently there aren’t lessons to learn from that project). The meeting would be good.
Formulating the evaluation questions around sustainable intensification (about sustainably intensified systems) would be great. What questions could we focus on to formulate evaluation questions around SI principles.
The landscape scale characterization (e.g. vegetation etc.) is useful e.g. from a project by ICRAF funded by DfID – this could be another layer we could include at a later stage. It’s a high priority for me.
In Ibadan we briefly discussed ethics and we identified the need to include text on ethics (e.g. privacy, dealing with farmers and not raising expectations etc.), language we can work from etc. that would be great to have in the research program on M&E and other topics.
We didn’t get to data flow protocol but that’s not going to be an easy topic. In CISSA it was not fully resolved. The principles in that project could inform ways that scientists get incentives to share data. IFPRI will have to follow a bottom up process so that IFPRI and others get data in a timely manner.
The program document – we developed a ToC but I couldn’t find it anywhere. That will be our product for September 30. In there we can have timelines explaining how the M&E framework will be more fully developed. There will be a chapter and timeline dedicated to M&E. We need a living program document by September 30, covering these issues. If we haven’t made sufficient progress we need to explain how we will make progress.
Q: We need clear objectives, types of activities, types of expected results to finalize the program document.
A: Yes, the program doc will give the general approaches and framework and we’ll have further details for specific projects, including research and development hypotheses, outcomes, M&E framework, ILRI comms strategy etc.
The research framework is about how – but we now need to focus on the ‘what we do’.
Comments:
For me to help with Africa RISING, I need Mali to be part of the M&E work. There are constraints from the US Gov’t. Logistics is not a good reason to cut in Mali. Cut at thematic level in terms of the types of technologies etc. that you will evaluate, not on geo boundaries.
The questionnaire needs to be checked by the research team before it’s finalized.
Thank you for the level of efforts of everyone over the last 2.5 days.